Method, apparatus, device and storage medium for evaluating quality of answer

ABSTRACT

Embodiments of the present disclosure provide a method, an apparatus, a device and a storage medium for evaluating quality of an answer. The method includes extracting a question feature expression of a question and an answer feature expression of an answer with respect to the question, the question and the answer being represented in a form of text; determining a measurement of textual quality of the answer based on the answer feature expression; determining a measurement of correlation on semantics between the question and the answer based on the question feature expression and the answer feature expression; and determining a quality score of the answer with respect the question based on the measurement of textual quality and the measurement of correlations. Therefore, a high-quality answer may be accurately obtained.

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority and benefits to Chinese Application No.201811521213.3, filed on Dec. 12, 2018, the entire content of which isincorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to a field of interaction technology, andmore particularly, to a method, an apparatus, a device and a storagemedium for evaluating quality of an answer.

BACKGROUND

With the development of network technology, more and more internetplatforms support a generation of user-generated content (UGC). Suchcontent includes social quiz, social comment, content sharing, and thelike. Questions and answers are natural forms for human to learnknowledge, as well as mediums for effectively exchanging and sharinginformation.

SUMMARY

Embodiments of the present disclosure provide a method for evaluatingquality of an answer. The method includes extracting a question featureexpression of a question and an answer feature expression of an answerwith respect to the question, the question and the answer beingrepresented in a form of text; determining a measurement of textualquality of the answer based on the answer feature expression;determining a measurement of correlation on semantics between thequestion and the answer based on the question feature expression and theanswer feature expression; and determining a quality score of the answerwith respect to the question based on the measurement of textual qualityand the measurement of correlation.

Embodiments of the present disclosure further provide an electronicdevice. The electronic device includes one or more processors and amemory configured to store one or more programs. When the one or moreprograms are executed by the one or more processors, the one or moreprocessors implement the above method.

Embodiments of the present disclosure further provide a computerreadable storage medium having computer program stored thereon. When theprogram is executed by the processor, the above method is implemented.

It is to be understood that the content of the present disclosure is notintended to limit the key or important characteristics of theembodiments, or the scope of the present disclosure. Additionalcharacteristics of the present disclosure will be readily understood bythe following description.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and/or additional features, aspects and advantages ofembodiments of the present disclosure become obvious and easilyunderstood in following descriptions with reference to accompanyingdrawings. Throughout the drawings, the same or similar reference numbersindicate the same or similar elements, in which:

FIG. 1 is a schematic diagram illustrating an exemplary environment forimplementing embodiments of the present disclosure.

FIG. 2 is a flowchart illustrating a method for evaluating quality of ananswer according to embodiments of the present disclosure.

FIG. 3 is a schematic diagram illustrating framework of a learningnetwork for evaluating quality of an answer according to embodiments ofthe present disclosure.

FIG. 4 is a block diagram illustrating an apparatus for evaluatingquality of an answer according to embodiments of the present disclosure.

FIG. 5 is a block diagram illustrating a computing device capable ofimplementing embodiments of the present disclosure.

DETAILED DESCRIPTION

Embodiments of the present disclosure will be described in more detailbelow with reference to the accompanying drawings. Although certainembodiments of the present disclosure are shown in the drawings, itshould be understood that the present disclosure may be embodied invarious forms and should not be construed to be limited to theembodiments set forth herein. The embodiments are provided to provide aclear and complete understanding of the present disclosure. It should beunderstood that, embodiments and drawings of the present disclosure aremerely exemplary and do not intent to limit the scope of the presentdisclosure.

In the description of the embodiments of the present disclosure, theterm “include” and the like are to be understood as open-ended, that is,“including but not limited to”. The term “based on” should be understoodas “at least partly based on”. The term “one embodiment” or “theembodiment” should be understood as “at least one embodiment”. The terms“first”, “second”, and the like may refer to different or identicalobject. Other explicit and implicit definitions may also be includedbelow.

With raising of economic level, people gradually pay attention to theimprovement of their self-knowledge level and take this improvement as alifelong learning task. Due to convenience of communication betweenpeople via the Internet, questions and answers on the interne platformhave become a convenient way for people to obtain information andknowledge. However, people also face a difficulty of screeninghigh-quality knowledge from a large number of answers.

In a knowledge-paid application, users can pay fees to ask a question toa trusted or professional person or institution to obtain an answer of ahigh quality. However, because there are a small number of trusted orprofessional people and institutions, a coverage of answers and thenumber of answers are limited. In addition to the knowledge-paidapplication that allows a small number of refined questions and answers,people can also achieve an economical and even free interaction ofquestion provision and answer acquisition on many social sites torealize a social interaction that allows a large number of all-embracingquestions and answers. Such sites often allow a user to provide ananswer, based on his knowledge and experience, to a question asked byanother user. In this way, a large amount of questions and answers maybe provided. However, due to the lack of a restriction mechanism, theanswers may be greatly deviated from real answers. For example, sinceanyone can post content that he/she wants to post, some users mayprovide an “answer” to a question for seeking entertainment or certainbenefit, independent of a real answer to the question. In addition,depending on the user's knowledge and expression ability, quality ofanswers varies with the users. Therefore, it is desirable to evaluatethe quality of an answer to distinguish a high-quality answer from alow-quality answer.

Therefore, it is desirable to evaluate the quality of an answer of aquestion to distinguish the high-quality answer from a low-qualityanswer. In a conventional solution, user voting may be used to obtainthe high-quality answer. For example, by providing a “LIKE” function foreach answer to encourage the user to vote on his/her satisfactory answerthrough this “LIKE” function, the high-quality answer may bedistinguished from other answers. However, this solution has somedrawbacks. Efficiency of this solution may be affected by a time factor.For example, on many websites, according to an existing mechanism, a newanswer provided by the user to a question may be displayed at the end ofan answer list. In most cases, users tend to browse the answer list fromthe top to the bottom. As a result, a possibility of seeing an answergenerated late in the time dimension may be lower than that of seeing ananswer generated early, such that the high-quality answer provided latemay be missed and the high-quality answer may obtain a low vote. Inaddition, this “all-embracing” mechanism for filtering answers is alsolimited by an influence of group psychology. It is observed that a highvoted answer is prone to get more votes, resulting in that each answermay not be feedback by all users fairly and objectively, especially forthe one post late.

Although some schemes are developed for automatically evaluating thequality of answers, these schemes rely largely on a large number oflinguistical analysis tools. Specifically, many schemes may usedifferent linguistical analysis tools to perform textual analysis onquestions and answers and manually select and extract features ondifferent levels, such as part-of-speech tagging, grammar, emotion,semantic features of the questions and answers. The manually selectedand extracted features may be used to train machine learningclassification models such as support vector machine (SVM) and randomforest. The trained models may be used to evaluate the quality of theanswers.

In general, conventional schemes for automatically evaluating valuesrely on the linguistical analysis tools, leading to many limitations.With many linguistical analysis tools, it is difficult to guarantee theaccuracy of analysis of target language. In addition, an applicationscenario with multi-type language may bring a cost of purchase anddevelopment on the linguistical analysis tools. Furthermore, whether thefeatures extracted via the linguistical analysis tools is advantageousor not to the evaluation of the quality of the answer is not settled.Even if some effective features may be defined by professors, theseschemes may be limited to a specific language environment, resulting ininsufficient universality. Therefore, these schemes often fail toaccurately find the high-quality answers.

With the in-depth study and application of deep learning, textualfeatures of the questions and answers may be extracted automatically bya neural network. These features may be used to train a pre-designedalgorithm model. The trained model may be used to obtain thehigh-quality answers. However, the inventors found that an existingneural network as a “black box” only focuses on a correlation betweenanswers and questions and outputs the quality of the answer accordingly.However, for some particular questions, an answer having a closecorrelation with the question is difficult to considered as ahigh-quality answer.

According to embodiments of the present disclosure, a solution forautomatically evaluating quality of an answer is provided. In thissolution, a question feature expression of the question and an answerfeature expression of the answer may be extracted. The answer featureexpression may be used to measure textual quality of the answer, and thequestion feature expression of the question and the answer featureexpression of the answer are used together to measure a correlation onsemantics between the answer and the question. A quality score of theanswer may be determined based on the textual quality and thecorrelation. In this way, the quality score of the answer with respectto the question may be measured from at least two aspects, i.e., thequality of the answer itself and the correlation between the answer andthe question, thereby accurately distinguishing the high-quality answerfrom other answers.

Embodiments of the present disclosure will be described in detail withreference to the drawings.

FIG. 1 is a schematic diagram illustrating an exemplary environment 100for implementing embodiments of the present disclosure. In the exemplaryenvironment 100, a computing device 102 may be configured to evaluatethe quality of an answer with respect to a certain question.

In embodiments of the present disclosure, the answer refers to aresponse to a certain question. The answer can also be called as amessage, a reply, a suggestion, a review, an opinion, and the like.Questions and answers may be usually written by users and submitted to aspecific website host. In embodiments of the present disclosure,discussions will be made based on comments given in a form of text. Insome cases, the comments may include content presented in a form ofaudio, video, pictures, and the like. For these situations, the contentin the form of audio, video and pictures may be converted to the form oftext for processing.

The computing device 102 may be configured to obtain a question and oneor more answers with respect to the question from a question-answerlibrary 104. Such a question-answer combination may be obtained from avariety of sources and may be presented or stored in anymanner/structure. For example, the question-answer combination may bepresented on a web page of a quiz website. In an example of FIG. 1, thecomputing device 102 may be configured to obtain a web page 110 from thequestion-answer library 104. The web page 110 may include a question 112and multiple answers 114-1, 114-2, and 114-3 (for ease of description,referred to as the answers 114) with respect to this question. It shouldbe understood that although multiple answers are shown, in some casesthere may be a single answer with respect to a question.

The computing device 102 may be configured to extract the question 112and the one or more answers 114 correspondingly from the web page 110and determine a quality score for each of the one or more answers 114with respect to the question 112 according to embodiments of the presentdisclosure. In embodiments of the present disclosure, the quality scoreof the answer 114 with respect to the question 112 may indicate thequality by taking the answer 114 as the answer to the question 112. Thecomputing device 102 may be configured to output a quality evaluationresult of the answer 114, i.e., the quality score determined.

The answers 114 may be categorized into different discrete categoriescorresponding to different quality levels, according to the qualityscores of the answers 114. For example, there are two quality levelsincluding a level of high-quality answers or a level of low-qualityanswers. In an example, there may be three or more other quality levels.The quality score may be configured to indicate the quality of theanswer 114 with a value within a continuous value range. The qualityscore of the answer 114 may be used for a variety of purposes. In anexample, the quality score of the answer 114 may be used to determine apresentation form of the answer 114 such that the high-quality answermay be highlighted or may be presented in a different manner from thatfor presenting the low-quality answer. FIG. 1 illustrates a case thatthe quality score of each of the answers 114 determines an order ofpresenting the answers 114. In detail, the computing device 102 may beconfigured to output the web page 120 based on the quality scores of theanswers 114. Compared with the web page 110, the computing device 102adjusts the order of presenting the answers 114 in the web page 120. Inthe web page 110, multiple answers 114 are presented from the top to thebottom in an order of posted time. Based on the quality scores of theanswers, the answer 114-3 that is posted late but has a high quality maybe presented at a upper portion of the web page 120, while the answer114-1 that is posted early but has a low quality may be presented at alower portion of the web page 120.

In addition to the presentation form of the answer 114, the qualityscore of the answer 114 may also affect various other decisions, such asthe award of the publisher of the answer 114, the promotion of theanswer 114 and/or the question 112. The scope of the embodiments of thepresent disclosure is not limited in this respect.

It should be understood that the web pages 110 and 120 illustrated inFIG. 1 are merely examples, and FIG. 1 illustrates a possibleapplication scenario according to embodiments of the present disclosure.In other embodiments, the computing device 102 may be configured toreceive textual content of questions and answers, rather than the webpages carrying the questions and answers, and provide the quality scoresof the answers. Such evaluation on the quality of the answer may beimplemented by an internet platform providing questions and answers, orby a third-party provider.

In order to clearly understand the method for evaluating quality of ananswer according to embodiments of the present disclosure, a detaileddescription will be made with reference to FIG. 2. FIG. 2 is a flowchartillustrating a method 200 for evaluating quality of an answer accordingto embodiments of the present disclosure. The method 200 may beimplemented by the computing device 102 illustrated in FIG. 1. For easeof discussion, the method 200 will be described with reference to FIG.1.

At block 210, the computing device 102 extracts a question featureexpression of a question 112 and an answer feature expression of ananswer 114 with respect to the question 112. The evaluation on a qualityscore of the answer 114 may be described below. The question 112 and theanswer 114 may be in a form of text. That is, the question 112 mayinclude text of one or more words, and the answer 114 may include textof one or more words. The text of the question 112 and the text of theanswer 114 may be represented in any same or different language.

For extracting features, the computing device 120 may be configured tovectorize the question 112 and the answer 114 to obtain a firstvectorized expression and a second vectorized expression. The questionfeature expression and the answer feature expression may be extractedbased on the first vectorized expression and the second vectorizedexpression.

For determining the vectorized expressions, the computing device 102 maybe configured to divide the questions 112 and the answers 114 based on acertain granularity to obtain one or more textual items, and determinethe vectorized expression corresponding to each of the textual items. Insome embodiments, the granularity used to obtain the textual items maybe related to the language of the text of the questions 112 and theanswers 114. For example, if the question 112 or the answer 114 includesa word spelled by Latin letters, such as English, French, German, thequestion 112 or the answer 114 may be divided based on a wordgranularity to obtain the textual items, such that each textual item mayinclude words contained in the question 112 or the answer 114. If thequestion 112 or the answer 114 includes pictographs, such as Chinese,Japanese, the question 112 or the answer 114 may be divided based on aphrase (or vocabulary) granularity, such that each textual item mayinclude a set of words (including one or more words) contained in thequestion 112 or the answer 114. For the text that is unable to bedivided by a specific identifier (such as a space), such as Chinese,Japanese, word segmentation tools may be used to divide the text toobtain the textual items. It may be appreciated that the question 112 orthe answer 114 may also be divided based on other granularities, asdesired, to obtain the textual items. The number of the textual itemsdepends on the specific content contained in the question 112 or theanswer 114.

The vectorized expression of the textual item may also be referred to asa vector coding of the textual item. Each vectorized expression of thetextual item may include multiple values of a certain dimension.Different vectorized expressions of the textual items may have a samedimension, but different values. A similarity between the vectorizedexpressions of the textual items may indicate a semantic similaritybetween different textual items. In embodiments of the presentdisclosure, by mapping the text to the vectorized expressions, influencecaused by difference among languages may be effectively reduced, toreduce application limitations. In some examples, the vectorizedexpressions of the textual items may be obtained from a predefinedvector coding set (codebook). The predefined codebook may be thevectorized expression of each textual item determined by encoding thetextual items included in the lexicon.

At block 220, the computing device 102 determines a measurement oftextual quality of the answer 114 based on the answer featureexpression. According to embodiments of the present disclosure, thetextual quality of the answer may be taken into account in response todetermining whether the answer 114 is a high-quality answer with respectto the question 112. The quality of the answer in a textual expressionaspect may affect the quality score of this answer. In some cases, ananswer may be highly correlated with the question, which solves thequestioner's doubts to some extents. However, since the answer is not ahigh-quality answer due to the low textual quality of the answer havingtext-related defects, such as, wrong words, grammatical errors, andredundant expressions. Such answer is less likely to be selected as thehigh-quality answer.

At block 230, the computing device 102 determines a measurement ofcorrelation on semantics between the answer 114 and the question 112based on the answer feature expression and the question featureexpression. A good answer is usually highly relevant to the question andmay provide a solution to the question, rather than providing anirrelative answer to the question. Therefore, the correlation onsemantics of the answer with respect to the question may also affect thequality score of the answer 114. At block 240, the computing device 102determines the quality score of the answer 114 with respect to thequestion 112 based on the measurement of textual quality and themeasurement of correlation on semantics. In embodiments of the presentdisclosure, a correspondence among the measurement of textual quality,the measurement of correlation on semantics and the quality score of theanswer with respect to the question may be established. Based on thecorrespondence, the measurement of textual quality and the measurementof correlation on semantics determined from a pair of the answer 114 andthe question 112 may be mapped to the quality score.

From the above, an overall solution for evaluating quality of an answeraccording to the embodiments of the present disclosure is provided. Bytaking both the measurement of textual quality and the measurement ofcorrelation on semantics between the answer and the question intoaccount, the evaluation on the quality of the answers is accurate,thereby facilitating to determine a high-quality answer satisfyingrequirements.

In some embodiments, extracting the feature expression, determining thetextual quality, determining the measurement of correlation, and/ordetermining the quality score as mentioned above may be implemented witha learning network. Therefore, the learning networks for implementingthe above processes may be integrated into an architecture of anend-to-end learning network. A main purpose of this learning network isto determine the quality score of the answer with respect to thequestion from entered questions and answers and use the quality score asan output.

FIG. 3 is a schematic diagram illustrating a framework of a learningnetwork 300 for evaluating quality of an answer according to embodimentsof the present disclosure. As used herein, the term “learning network”refers to a model that is capable of learning, from training data, acorrelation between inputs and outputs, thereby processing a given inputbased on a set of trained parameters after the training is completed togenerate a corresponding output. The “learning network” may also bereferred to as “neural network”, “learning model”, “network”, or“model”. These terms may be used interchangeably herein.

Operations of the learning network 300 illustrated as FIG. 3 may beimplemented by the computing device 102 of FIG. 1. For ease ofdescription, the exemplary architecture of FIG. 3 may be described withreference to FIG. 1. It is assumed that in the learning network 300, theset of parameters used by the model for processing are obtained throughthe training and may be used for evaluating the quality of the answer.As illustrated in drawings, the learning network 300 may include afeature extracting model 310 configured to extract a question featureexpression of a question, a feature extracting model 320 configured toextract an answer feature expression of an answer, an importanceevaluating model 330 configured to evaluate textual quality of theanswer, a correlation evaluating model 340 configured to evaluate acorrelation between the answer and the question, and a qualityevaluation model 350 configured to evaluate the quality.

In usage, the computing device 102 is configured to obtain a vectorizedexpression 312 (denoted as Q, sometimes also referred to as a firstvectorized expression) of the question 112 as an input of the featureextraction model 310. For example, the question 112 is divided into ntextual items, the vectorized expression 312 may include vector codingcorresponding to each of then textual items, that is, Q=q₁,q₂,q₃, . . .,q_(n) where Q ∈

^(n×d), q_(k) represents the vector coding of a k^(th) textual item ofthe question 112, q_(k) ∈

^(d), d represents a dimension of the vector coding, i.e., each textualitem may be represented by a vector composed of d elements. The featureextraction model 310 may be configured to extract the question featureexpression 314 (denoted as {tilde over (Q)}) from the vectorizedexpression 312, where {tilde over (Q)}=({tilde over (q)}₁,{tilde over(q)}₂,{tilde over (q)}₃, . . . ,{tilde over (Q)}_(n),), {tilde over (Q)}∈

^(n×h), {tilde over (q)}_(k) represents a k^(th) vector coding extractedby the character extraction model 310 and corresponding to the kthtextual item of the question 112, {tilde over (q)}_(k) ∈

^(h), h is a dimension of the vector coding extracted by the featureextraction model 310 and is related to a structure of the model 310.

The feature extraction model 310 may be constructed as multiple types ofneural network models, as long as it may be used to extract textualfeatures. In some embodiments, the feature extraction model 310 may be along short-term memory (LSTM) model, also referred to as a first LSTMmodel. FIG. 3 illustrates that the feature extraction model 310 isimplemented as the LSTM model. Thus, {tilde over (Q)}=({tilde over(q)}₁,{tilde over (q)}₂,{tilde over (q)}₃, . . . , {tilde over(q)}_(n),)=LSTM(Q)=LSTM(q₁,q₂,q₃, . . . ,q_(n)). The LSTM model has aset of trained parameters and is configured to map the vectorizedexpression Q of the answer 114 to the answer feature expression {tildeover (Q)}. In an embodiment employing the LSTM model, the dimension h ofeach vector coding of the question feature expression 314 may correspondto the number of neurons of a hidden layer in the LSTM model.

With the LSTM model, a sequential correlation between the textual itemand other textual items in the question 112 may be taken into accountduring extracting features of each textual item. For example, thecorrelation between the textual item and previous one or more textualitems or the correlation between the textual item and later one or moretextual items may be taken into account, such that contextual semanticsof the question 112 may be considered. Thus, the extracted questionfeature expression 314 and the contextual semantics of the question 112may be used together to accurately characterize the question 112. Itshould be understood that the LSTM model is an example. The featureextraction model 310 may be another model for extracting features, suchas a recursion neural network, a convolutional neural network, and thelike. The scope of embodiments of the present disclosure is not limitedin this respect.

Similarly, the computing device 102 may be configured to obtain avectorized expression of the answer 114 to be evaluated as an input ofthe feature extraction model 320. For a single question 112, if thereare multiple answers 114 (for example L answers) to be evaluated, thevectorized expression of each answer may be sequentially input into thefeature extraction model 320 to extract the corresponding featureexpression. Only a vectorized expression 314 of the answer 114 (denotedas A^(i), sometimes also referred to as the second vectorizedexpression) is illustrated in FIG. 3. For example, the answer 114 isdivided into m textual items, and the vectorized expression 114 includesvector coding corresponding to the m textual items. That is, A^(i)=(a₁^(i),a₂ ^(i),a₃ ^(i), . . . ,a_(m) ^(i)), where A^(i) ∈

^(m×d), and a_(j) ^(i) ∈

^(d), where, d represents a dimension of the vector coding, i.e., eachtextual item may be represented by a vector composed of d elements.

The feature extraction model 320 may be configured to extract the answerfeature expression (denoted as Ã^(i)) from the vectorized expression ofthe input answer 114, where Ã^(i)=(ã₁ ^(i),ã₂ ^(i),ã₃ ^(i), . . . ,ã_(m)^(i)), Ã^(i) ∈

^(m×h), ã_(k) ^(i), represents a k^(th) vector coding extracted by thefeature extraction model 320 and corresponding to ae k^(th) textual itemin the answer 114, ã_(k) ^(i) ∈

^(h), h is a dimension of the vector coding extracted by the featureextraction model 320 and is related to a structure of the model 320.FIG. 3 illustrates that the answer feature expression 324 is extractedfrom the vectorized expression 314 of the answer 114 (Ã¹), where Ã¹=(ã₁¹,ã₂ ¹,ã₃ ¹, . . . ,ã_(m) ¹). For each answer, such as each of Ã² toÃ^(L), the answer feature expression may be extracted similarly. Itshould be understood that although the dimension of ã_(k) ^(i) is setherein to be same as the dimension of {tilde over (q)}_(k), in otherexamples, depending on the configurations of the feature extractionmodels 310 and 320, the dimension of ã_(k) ^(i) may be different fromthe dimension of {tilde over (q)}_(k). Hereinafter, for convenience ofdescription, the description is made where the dimensions of ã_(k) ^(i)and {tilde over (q)}_(k) are same.

The feature extraction model 320 may be constructed as multiple neuralnetwork models, as long as it may be used to extract textual features.In some embodiments, the feature extraction model 320 may be a LSTMmodel which may be also referred to as a second LSTM model. FIG. 3illustrates that the feature extraction model 320 is implemented as theLSTM model. Thus, Ã^(i)=(ã₁ ^(i),ã₂ ^(i),ã₃ ^(i), . . . ,ã_(m)^(i))=LSTM(Ã^(i))=LSTM(a₁ ^(i),a₂ ^(i),a₃ ^(i), . . . ,a_(m) ^(i)). TheLSTM model has a set of trained parameters and is configured to map thevectorized expression A^(i) of the answer 114 to the answer featureexpression Ã^(i). In an embodiment where the LSTM model is used, thedimension h of each vector coding of the answer feature expression 324may correspond to the number of neurons of the hidden layer in the LSTMmodel.

With the LSTM model, the contextual semantics of the textual item of theanswer 114 may be taken into account during extracting features of eachtextual item. The extracted answer feature expression 324 and thecontextual semantics of the answer 114 may be used together toaccurately characterize the answer 114. It should be understood that theLSTM model is an example. The feature extraction model 320 may be othermodels for extracting features, such as a recurrent neural network, aconvolutional neural network, and the like. The scope of embodiments ofthe present disclosure is not limited in this respect. The featureextraction model 310 and the feature extraction model 320 may beseparated models and may be individually trained and have the set ofparameters respectively. The feature extraction models 310 and 320 maybe different when being constructed as the LSTM model.

In some embodiments, in order to determine the textual quality of theanswer 114, the computing device 102 may be configured to apply a“single step attention mechanism” to focus on or highlight features ofimportant textual items in the answer 114, while ignoring features ofunimportant textual items. This may be achieved by the importanceevaluating model 330. In detail, the importance evaluating model 330 maybe configured to determine an importance of each textual item in theanswer 114 in the context of the answer 114. The importance evaluationmodel 330 may be configured to perform an importance evaluation based onthe answer feature expression 324. The importance is used such that thesemantic features of a subset of textual items having a high importancemay provide a high contribution to a result of evaluation on the qualityof the answer, while semantic features that are less effective may havea small impact on the evaluation of the quality of the answer. In thisway, the accuracy of the quality evaluation result may be improved.

In operation, the importance evaluation model 330 has a set of trainedparameters and is configured to map each feature element, from theanswer feature expression (e.g., the answer feature expression 324),corresponding to each textual item of the answer 114, such as ã₁ ^(i),ã₂^(i),ã₃ ^(i), . . . ,ã_(m) ^(i), to a value range of the importance. Forexample, an activation function of the importance evaluation model 330may be a Tanh activation function. The result may be mapped to aspecific value range using Softmax function. The processing of theimportance evaluation model 330 may be represented as follows:

γ_(j) ^(i)=softmax((v ^(A))^(T) tanh(W ^(A) ã _(j) ^(i)))   (1)

where

${{\tanh (x)} = \frac{e^{x} - e^{- x}}{e^{x} + e^{- x}}},{{{and}\mspace{14mu} {{softmax}( z_{j} )}} = {\frac{e^{z_{j}}}{\sum\limits_{k = 1}^{K}e^{z_{k}}}.}}$

In the formula (1), γ_(j) ^(i) represents the importance of a j^(th)textual item in the context of an i^(th) answer. The set of parameters332 W^(A) and the set of parameters 334 v^(A) may be used by theimportance evaluation model 330 to map the inputted answer featureexpression to an output of an important degree, where v^(A) ∈

^(h) and W^(A) ∈

^(h×h). The importance of each textual item of each answer in thecontext of the answer may be determined. FIG. 3 illustrates anexpression 336 of the importance of each textual item of answer A¹ basedon the answer feature expression 324.

The computing device 102 may be configured to determine a measurement oftextual quality of the answer 114 by weighting the feature elements inthe answer feature expression with the importance, which may beexpressed as follows:

$\begin{matrix}{x^{A^{i}} = {\sum\limits_{j}{\gamma_{j}^{i}{\overset{\sim}{a}}_{j}^{i}}}} & (2)\end{matrix}$

where, x^(A) ^(i) represents the measurement of textual quality of thei^(th) answer, and x^(A) ^(i) ∈

^(h). FIG. 3 illustrates weighting the answer feature expression 324using the importance to determine the measurement x^(A) ¹ of the textualquality 338 of the answer 114 A¹.

It should be understood that although the processing of the importanceevaluation model 330 is described above by taking the Tanh activationfunction and the Softmax normalized output function as an example, inother examples, other types of activation functions and output functionsmay be employed by the importance evaluation model 330. The scope ofembodiments of the present disclosure is not limited in this respect.

The question feature expression 314 and the answer feature expression ofeach answer 114 (e.g., the answer feature expression 324) may beprovided to the correlation evaluation model 340 for evaluating thecorrelation between the answer 114 and the question 112. The questionfeature expression of the question 112 and the answer feature expressionof the answer 114 may characterize the question 112 and the answer 114respectively to some extent. Therefore, the correlation evaluation model340 may be configured to determine whether the question 112 and theanswer 114 are semantically related, i.e., determine the semanticmatching or similarity.

In some embodiments, the correlation evaluation model 340 may beconfigured to determine a set of item-level correlations between theanswer 114 and the question 112 on a level of textual item based on thequestion feature expression 314 and the answer feature expression (e.g.,the answer feature expression 324). The set of item-level correlationsincludes elements arranged in rows and columns. Each element may be usedto indicate the correlation of one textual item of the answer withrespect to a textual item of the question. To determine each element ofthe set of item-level correlations, the correlation evaluation model 340may be configured to link the question feature expression 314 to afeature element corresponding to each textual item in the answer featureexpression, and map the linked feature element to a correspondingcorrelation. This may be expressed as follows:

c _(j,k) ^(Q,A) ^(i) =tanh((ã _(j) ^(i) ⊕{tilde over (q)} _(k))^(T) u)  (3)

where, the tanh function is similar to that described in the formula(1), ã_(j) ^(i) represents the vector coding corresponding to the j^(th)textual item of the answer in the answer feature expression of thei^(th) answer 114, and {tilde over (q)}_(k) represents a vector codingcorresponding to the k^(th) textual item of the question in the questionfeature expression of the question 112, ⊕ represents linking/jointing ofthe vector coding, and c_(j,k) ^(Q,A) ^(i) represents the correlationbetween the k^(th) textual item of the question and the j^(th) textualitem of the answer. The correlation evaluation model 340 is configuredto map the linked vector coding to the corresponding correlation usingthe set 343 u of parameters, where u ∈

^(2h).

For illustrative purposes, FIG. 3 illustrates that the feature elementã₁ ¹ corresponding to the first textual item of the answer in the answerfeature expression 324 is jointed with the feature element {tilde over(q)}₁ corresponding to the first textual item in the question featureexpression 314 to obtain a jointed feature element 342. The jointedfeature element 342 may be mapped using the set 343 u of parameters toobtain an element at an intersection of the first row and the firstcolumn of the set 345 of item-level correlations (denoted as C^(Q,A) ¹ )between the question 112 Q and the answer 114 A¹. The correlation amongeach textual item in the question 112 and in each answer 114 may bedetermined, thereby forming a set of item-level correlations C^(Q,A)^(i) between the question 112 and the answer 114.

The correlation evaluation model 340 may be configured to weight theanswer feature expression and the question feature expression using theset of item-level correlations to determine a measurement of correlationbetween the question 112 and the answer 114. In order to take featuresof the textual item having a high correlation into account, in someembodiments, based on the set of item-level correlations, thecorrelation evaluation model 340 may be configured to identify a vectorencoding that is closely relevant to the answer 114 from the questionfeature expression 314 and identify a vector encoding that is closelyrelevant to the question 112 from the answer feature expression (e.g.,the answer feature expression 324).

Depending on an arrangement of rows and columns of the set of item-levelcorrelations, each row of the set of item-level correlations mayindicate the correlation of each textual item of the answer 114 withrespect to the question 112, while each column may indicate thecorrelation of each textual item of the question 112 with respect to theanswer 114, as illustrated in the set 345 of item-level correlationsC^(Q,A) ¹ in FIG. 3. In an example, in another arrangement, each row ofthe set of item-level correlations may indicate the correlation of eachtextual item of the question 112 with respect to the answer, while eachcolumn may indicate the correlation of each textual item of the answer114 with respect to the question 112.

Thus, in order to identify vector coding with a high correlation fromthe question feature expression and the answer feature expression,elements with a high correlation (e.g., element having a correlationhigher than a first threshold and a second threshold) may be selectedrow by row and column by column from the set of item-level correlations.In other words, for each row of the set of item-level correlations, theelements having a high value in the row may be sequentially selected.For each column of the set of item-level correlations, the elementshaving a high value in the column may be sequentially selected. In thismanner, a first subset (denoted as β^(Q)) of significant correlations ofthe question 112 with respect to the answer 114 and a second subset(represented as β^(A) ^(i) ) of significant correlations of the answer114 with respect to the question 112 may be generated. In the example ofFIG. 3, for the question 112 Q and the answer 114 A¹, by selecting theelements with a high value column by column, the first subset 346 ofsignificant correlations may be determined, and by selecting theelements with a high value row-by-row to determine the second subset 347of significant correlations.

In the selection of the elements with a higher value column by columnand row-by-row, in some embodiments, an element with a maximum value maybe selected from one row and/or one column each time. This is calledrow-by-row maximum pooling and column-by-column maximum pooling. In theexemplary arrangement of FIG. 3, by performing the maximum pooling onthe set of item-level correlations by rows (i.e., portrait orientation)and by normalizing and performing probability distribution standardizingusing the Softmax function, the first subset of significant correlationsof the question 112 relative to the answer 114 may be determined, whichmay be denoted as follows:

β^(Q)=softmax(max([c _(:,1) ^(Q,A) ^(i) ,c _(:,2) ^(Q,A) ^(i) ,c _(:,3)^(Q,A) ^(i) , . . . ,c _(:,n) ^(Q,A) ^(i) ]))   (4)

in a similar manner, the second subset of significant correlations ofthe answer 114 relative to the question 112 may be determined, which maybe denoted as follows:

β^(A) ^(i) =softmax(max([c _(1,:) ^(Q,A) ^(i) ,c _(2,:) ^(Q,A) ^(i) ,c_(3,:) ^(Q,A) ^(i) , . . . ,c _(m,:) ^(Q,A) ^(i) ]))   (5)

where, the Softmax function in equations (4) and (5) is similar to thatdescribed in formula (1).

The first subset of significant correlations may include elements, inthe set of item-level correlations, indicating a high correlationbetween the question 112 and the answer 114 on the textual item level,while the second subset of significant correlations may includeelements, in the set of item-level correlations, indicating a highcorrelation between the answer 114 and the question 112 on the textualitem level. The computing device 102 may be configured to utilize thefirst subset of significant correlation to weight the question featureexpression 314 and utilize the second subset of significant correlationsto weight the answer feature expression (e.g., the answer featureexpression 324), thereby highlighting the question features and theanswer features having the high correlations. The weighted questionfeature expression and the weighted answer feature expression are linkedto generate a measurement of correlation between the question 112 andthe answer 114. Weighting and linking the question feature expressionand the answer feature expression may be denoted as follows:

$\begin{matrix}{x^{Q,A^{i}} = {( {\sum\limits_{k}{\beta_{k}^{Q}q_{k}}} ) \oplus ( {\sum\limits_{j}{\beta_{j}^{A^{i}}a_{j}^{i}}} )}} & (6)\end{matrix}$

where x^(Q,A) ^(i) represents the measurement of correlation between thequestion 112 and the i^(th) answer 114. FIG. 3 illustrates the weightedquestion feature expression 348 of the question 112 and the weightedanswer feature expression 349 of the answer 114 A¹. These two featureexpressions are linked to generate the measurement of correlation 352between the question 112 and the answer 114 A¹.

In embodiments of the present disclosure, for the question 112 and eachof the answers 114 with respect to the question, the determinedmeasurement of textual quality and the measurement of correlation areused together to determine the quality score for the current answer 114.In detail, for each question 114, the measurement of textual quality andthe measurement of correlation are linked as an input of the qualityevaluation model 350, which may be expressed as follows:

x ^(i) =x ^(Q,A) ^(i) ⊕x ^(A) ^(i)   (7)

FIG. 3 illustrates an input x¹ obtained by linking the measurement oftextual quality 338 of the answer 114 A¹ and the measurement ofcorrelation 352 between the question 112 and the answer 114 A¹. Thequality evaluation model 350 may be configured to determine the qualityscore of the answer 114 A^(i) relative to the question 112 based on theinput x^(i). The quality evaluation model 350 may be configured to mapthe input x^(i) to the corresponding quality score using the set oftrained parameters.

In some embodiments, if there are multiple answers 114 with respect tothe question 112, these answers 114 may be generated in an order or maybe released in an order of the questions 112. These answers 114 may forma sequence based on any order, such as posting time, currentpresentation order, and the like. In the case of multiple answers,different answers may influence each other. For example, if aprior-ranked answer is determined to be a high-quality answer, there isa low probability that the subsequent similar answer is considered as ahigh-quality answer, thereby avoiding repeated presentation of theanswers. In some cases, the answer generated later may be more likely torefer to the previous answer, and thus the probability of selecting theanswer generated later as a high-quality answer may be high. If ananswer contains content of multiple previous answers, the probabilitythat the answer is a high-quality answer is high. Thus, in the case ofmultiple answers, the result of evaluation on the quality of otheranswers may affect the evaluation of the current answer.

In some embodiments, for a given answer 114, the quality evaluationmodel 350 may be configured to determine a candidate quality score ofthe answer 114 with respect to the question 112 based on the measurementof textual quality and the measurement of correlation, and adjust thecandidate quality score based on the quality scores of other answers inthe sequence, particularly the reference quality scores of other answersranked before the current answer 114. The quality evaluation model 350may be configured to determine the quality score of each answersequentially. For the current answer 114, if there are other answersbefore the answer, the quality evaluation model 350 may be configured toobtain the determined quality scores of these answers as the referencequality scores. The quality evaluation model 350 may be also configuredto obtain the probability of converting the reference quality score tothe candidate quality score in a sequence comprising multiple orderedanswers. The quality score may be determined based on the candidatequality score and the determined probability.

For other answers in the sequence, the quality evaluation model 350 maybe configured to similarly determine the corresponding quality score. Insome embodiments, the determination of the quality score may beimplemented using a conditional random field (CRF) model. In such anembodiment, the determination of the quality score may be expressed asfollows:

$\begin{matrix}{{{\Pr( y^{1:L} \middle| x^{A^{Q,A^{1:L}}} )} \propto {\exp( {{\sum\limits_{t}{O\lbrack {y^{t - 1},y^{t}} \rbrack}} + {\sum\limits_{t}{g^{t}\lbrack y^{t} \rbrack}}} )}}{{where},{{g^{t}\lbrack y^{t} \rbrack} = {\log ( {{W^{T}x^{t}} + b} )}}}} & (8)\end{matrix}$

where Pr(y^(1:L)|x^(A) ^(1:L) ,x^(Q,A) ^(1:L) ) represents the qualityscore of each of the L answers 114 with respect to the question 112 or aquality classification y^(t), the set of parameters W^(T) ∈

^(3h×z), b ∈

^(z), z indicates the categories of the quality classifications of theanswer (for example, including two categories, i.e., a category ofhigh-quality answers and a category of low-quality answers). 0 ∈

^(z×z) represents a transfer matrix, and 0[y^(t−1),y^(t)] represents aprobability or a weight of converting the quality score y^(t−1) of theanswer t−1 to the quality score y^(t) of the answer t, where the weightmay be set in advance. As can be seen from formula (8), the qualityscores between related answers may affect each other. By consideringthis effect, it is possible to accurately determine subsequenthigh-quality answers in a case where there are multiple answers, withoutcausing repetition of high-quality answers.

A process that the learning network 300 is used to determine the qualityof the answer in a case where the set of parameters of the learningnetwork 300 is trained is described above. A process of training thelearning network 300 will be described below. The purpose of thetraining is to continuously optimize the set of parameters of thelearning network from the initial value to achieve a certain convergencecondition (i.e., a learning objective). Training the learning network300 can be implemented by the computing device 102. In an example, thelearning network 300 may be trained by devices other than the computingdevice 102 and may be used by the computing device 102.

In the learning network 300 of FIG. 3, the set of parameters that needsto be trained includes: a set of parameters of the feature extractionmodels 310 and 320, the sets v^(A) and W^(A) of parameters of thetextual quality evaluation model 330, the set u of parameters of thecorrelation evaluation model 340, and the sets W^(T) and b of parametersof the quality evaluation model 350. In the learning network 300, someparameters may be automatically or manually set to be fixed, such as thedimension d of the vector expression of the textual item, the length nof the question text, the length m of the answer text, and the like.These parameters may be referred to as hyperparameters.

Training data used in training the learning network 300 may includemultiple training questions, one or more answers for each trainingquestion, and true labels of these answers, i.e., labels indicatingwhether the answer is a high-quality answer or a quality score of theanswer. At the beginning of the training processing, the set ofparameters of the learning network 300 may be initialized. Randominitialization may be used to set the initial value of each parameter.In some examples, for some parameters or models, pre-trained parametervalues may be obtained as initial values.

In some embodiments, the vectorized expression of the textual itemand/or a sub textual item may also be considered as a parameter to beoptimized. In this way, the vectorized expression that is useful todetermine the evaluation on the quality of the answer in the context ofthe evaluation on the quality of the answer may be determined throughthe training process. While optimizing the sets of parameters of variousportions of the learning network 300, the vectorized expressions of thetextual items and/or sub textual items may be also constantly updatedand optimized to determine the final vectorized expression.

During the training process, multiple learning algorithms (e.g., astochastic gradient descent algorithm, a back propagation algorithm) maybe used to continuously optimize the set of parameters of the learningnetwork 300, such that an error between a predicted degree and a targetdegree may be continuously reduced, and thus a value of a loss functionmay be continuously decreased. The loss function may be expressed asΣ−y^(1:L) log(Pr(y^(1:L)|x^(A) ^(1:L) ,x^(Q,A) ^(1:L) )). If the lossfunction is convergent (e.g., the error is within a predeterminedthreshold range, or a value of the loss function is decreased to a smallvalue or is minimized, or a predetermined number of iterations isreached) or the number of iterations reaches a preset number, thecurrent parameter value of the learning network 300 may be determined asthe set of parameters after the training.

According to embodiments of the present disclosure, an efficient andaccurate automatic evaluation on the quality of the answer may beachieved, and an accurate answer may be determined by taking aninteraction among different answers into account in a case of multipleanswers.

FIG. 4 is a block diagram illustrating an apparatus 400 for evaluatingquality of an answer according to embodiments of the present disclosure.The apparatus 400 may be included in the computing device 102 of FIG. 1or implemented as the computing device 102. As illustrated in FIG. 4,the apparatus 400 includes a feature extraction module 410, a textualquality evaluation module 420, a correlation evaluation module 430 andan answer quality evaluation module 440. The feature extraction module410 may be configured to extract a question feature expression of aquestion and an answer feature expression of an answer with respect tothe question. The question and the answer are in a form of text. Thetextual quality evaluation module 420 may be configured to determine ameasurement of textual quality of the answer based on the answer featureexpression. The correlation evaluation module 430 may be configured todetermine a measurement of correlation on semantics between the questionand the answer based on the question feature expression and the answerfeature expression. The answer quality evaluation module 440 may beconfigured to determine a quality score of the answer with respect tothe question based on the measurement of textual quality and themeasurement of correlation on semantics.

In some embodiments, the answer may be divided into multiple textualitems and the answer feature expression may include feature elementscorresponding to the textual items. The textual quality evaluationmodule 420 may include an importance determination module, configured todetermine an importance of each textual item in the context of theanswer. In addition, the textual quality evaluation module 420 mayfurther include a weight-based quality evaluation module, configured todetermine the measurement of textual quality by weighting the featureelements in the answer feature expression using the importance.

In some embodiments, the correlation evaluation module 430 may include atextual item correlation determination module, configured to determine aset of item-level correlations between the answer and the question on atext-item level based on the answer feature expression and the questionfeature expression. The set of item-level correlations may includeelements arranged in rows or columns. Each element may be used toindicate the correlation between a textual item of the answer and atextual item of the question. In addition, the correlation evaluationmodule 430 may further include a weight-based correlation evaluationmodule 430, configured to determining the measurement of correlation byweighting the answer feature expression and the question featureexpression using the set of item-level correlations.

In some embodiments, the weight-based correlation evaluation module 430may include a correlation selection module, a question feature weightingmodule, an answer feature weighting module and a linking module. Thecorrelation selection module may be configured to generate a firstsubset of significant correlations of the question with respect to theanswer and a second subset of significant correlations of the answerwith respect to the question by selecting elements having a higher valuethan a predetermined threshold row by row and column by column, from theset of item-level correlations. The question feature weighting modulemay be configured to weight the question feature expression using thefirst subset of significant correlations. The answer feature weightingmodule may be configured to weight the answer feature expression usingthe second subset of significant correlations. The linking module may beconfigured to link the weighted question feature expression and theweighted answer feature expression to obtain the measurement ofcorrelation.

In some embodiments, the correlation selection module may include amaximum pooling module, configured to generate the first subset ofsignificant correlations and the second subset of significantcorrelations by applying maximum pooling on the set of item-levelcorrelations row by row and column by column.

In some embodiments, the answer may be included in a sequence includingmultiple ordered answers with respect to the question. The answerquality evaluation module 440 may include: a candidate qualitydetermination module, a reference quality obtaining module, a conversionprobability determination module and a combination quality evaluationmodule. The candidate quality determination module may be configured todetermine a candidate quality score based on the measurement of textualquality and the measurement of correlation. The reference qualityobtaining module may be configured to obtain a reference quality scoreof the answer ranked before the answer in the sequence. The conversionprobability determination module may be configured to determine aprobability of converting the reference quality score to the candidatequality score with respect to the sequence. The combination qualityevaluation module may be configured to determine the quality score basedon the candidate quality score and the determined probability.

In some embodiments, the answer quality evaluation module 440 may beconfigured to determine the quality score using a trained conditionalrandom field (CRF) model.

In some embodiments, the feature extraction module 410 may include: afirst model extraction module and a second model extraction module. Thefirst model extraction module may be configured to extract the questionfeature expression using a first long-short term memory (LSTM) modeltrained. The second model extraction module may be configured to extractthe answer feature expression using a second LSTM model trained. Thesecond LSTM model is different from the first LSTM model.

In some embodiments, the feature extraction module 410 may include avectorization module, a question feature extraction module and an answerfeature extraction module. The vectorization module may be configured toacquire a first vectorized expression of the question and a secondvectorized expression of the answer. The question feature extractionmodule may be configured to extract the question feature expressionbased on the first vectorized expression. The answer feature extractionmodule may be configured to extract the answer feature expression basedon the second vectorized expression.

FIG. 5 is a block diagram of an example device 500 capable ofimplementing embodiments of the present disclosure. The device 500 maybe used to implement the computing device 102 of FIG. 1. As shown, thedevice 500 may include a computing unit 501 that may perform variousappropriate actions and processes according to computer programinstructions stored in a read-only memory (ROM) 502 or computer programinstructions loaded from the storage unit 508 to a random access memory(RAM) 503. In the RAM 503, various programs and data required for theoperation of the device 500 can also be stored. The computing unit 501,the ROM 502, and the RAM 503 may be connected to each other through abus 504. An input/output (I/O) interface 505 may be also coupled to bus504.

A plurality of components in the device 500 are coupled to the I/Ointerface 505, including: input units 506, such as keyboard, mouse;output units 507, such as various types of displays, speakers; storageunits 508, such as disks, optical disks; and communication units 509such as network cards, modems, wireless communication transceivers. Thecommunication unit 509 allows the device 500 to exchangeinformation/data with other devices over a computer network such as theInternet and/or various telecommunication networks.

The computing unit 501 can be a variety of general purpose and/orspecial processing components with processing and computingcapabilities. Some examples of the computing unit 701 include, but arenot limited to, central processing unit (CPU), a graphics processingunit (GPU), various specialized artificial intelligence (AI) computingchips, various computing units that run machine learning modelalgorithms, digital signal processor (DSP), and any suitable processor,controller, microcontroller. The computing unit 501 can perform thevarious methods and processes described above, such as the process 200.For example, in some embodiments, the process 200 can be implemented asa computer software program that is tangibly embodied in a machinereadable medium, such as the storage unit 508. In some embodiments, someor all of the computer program may be loaded and/or installed onto thedevice 500 via the ROM 502 and/or the communication unit 509. One ormore steps of the process 200 described above may be performed when acomputer program is loaded into the RAM 503 and executed by thecomputing unit 501. Alternatively, in other embodiments, the computingunit 501 can be configured to perform the process 200 by any othersuitable means (e.g., by means of firmware).

The functions described above herein may be performed, at least in part,by one or more hardware logic components. By way of example but notlimitation, exemplary types of hardware logic components that may beused include: field programmable gate array (FPGA), application specificintegrated circuit (ASIC), application specific standard product (ASSP),system on a chip (SOC), complex programmable logic device (CPLD).

Program code for implementing the methods of the present disclosure canbe written in any combination of one or more programming languages. Theprogram code may be provided to a general purpose computer, a specialpurpose computer or a processor or controller of other programmable dataprocessing device such that the program code, when executed by theprocessor or controller, causes the functions/operations specified inthe flowcharts and/or block diagrams to be implemented. The program codemay execute entirely on the machine, partly on the machine, as part ofthe stand-alone software package, and partly on the remote machine orentirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium canbe a tangible medium that can contain or store a program for use by orin combination with an instruction execution system, apparatus, ordevice. The machine readable medium can be a machine readable signalmedium or a machine readable storage medium. A machine-readable mediumcan include, but is not limited to, an electronic, magnetic, optical,electromagnetic, infrared, or semiconductor system, apparatus, ordevice, or any suitable combination of the foregoing. More specificexamples of machine readable storage medium may include electricalconnections based on one or more wires, a portable computer disk, a harddisk, a random access memory (RAM), a read only memory (ROM), anerasable programmable read only memory (EPROM or flash memory), anoptical fiber, a convenient compact disk read only memory (CD-ROM), anoptical storage device, a magnetic storage device, or any suitablecombination of the foregoing.

In addition, although the operations are depicted in a particular order,this should be understood to require that such operations be performedin the particular order shown or in sequential order, or that allillustrated operations should be performed to achieve the desiredresults. Multitasking and parallel processing may be advantageous incertain circumstances. Likewise, although several specificimplementation details are included in the above discussion, theseshould not be construed as limiting the scope of the disclosure. Certaincharacters that are described in the context of separate embodiments canalso be implemented in combination in a single implementation.Conversely, various characters that are described in the context of asingle implementation can be implemented in a plurality ofimplementations, either individually or in any suitable sub-combination.

Although the subject matter has been described in language specific tostructural characters and/or methodological acts, it is understood thatthe subject matter defined in the appended claims is not limited to thespecific features or acts described above. Instead, the specificcharacters and acts described above are merely exemplary forms ofimplementing the claims.

What is claimed is:
 1. A method for evaluating quality of an answer,comprising: extracting a question feature expression of a question andan answer feature expression of an answer with respect to the question,the question and the answer being represented in a form of text;determining a measurement of textual quality of the answer based on theanswer feature expression; determining a measurement of correlation onsemantics between the question and the answer based on the questionfeature expression and the answer feature expression; and determining aquality score of the answer with respect to the question based on themeasurement of textual quality and the measurement of correlation. 2.The method of claim 1, further comprising: dividing the answer into aplurality of textual items, the answer feature expression comprisingfeature elements corresponding to the plurality of textual items,wherein, determining the measurement of textual quality comprises:determining an importance of each textual item in context of the answer;and determining the measurement of textual quality by weighting thefeature elements of the answer feature expression with the importance.3. The method of claim 1, wherein determining the measurement ofcorrelation comprises: determining a set of item-level correlationsbetween the answer and the question on a level of textual item based onthe answer feature expression and the question feature expression, theset of item-level correlations comprising elements arranged in rows andcolumns, each element being configured to indicate a correlation betweena textual item of the answer and a textual item of the question; andweighting the answer feature expression and the question featureexpression using the set of item-level correlations to determine themeasurement of correlation.
 4. The method of claim 3, wherein weightingthe answer feature expression and the question feature expressioncomprises: generating a first subset of significant correlations of thequestion with respect to the answer and a second subset of significantcorrelations of the answer with respect to the question by selecting anelement having a higher value than a predetermined threshold row by rowand column by column, from the set of item-level correlations; weightingthe question feature expression using the first subset of significantcorrelations; weighting the answer feature expression using the secondsubset of significant correlations; and linking the question featureexpression weighted and the answer feature expression weighted to obtainthe measurement of correlation.
 5. The method of claim 4, whereingenerating the first subset of significant correlations and the secondsubset of significant correlations comprises: generating the firstsubset of significant correlations and the second subset of significantcorrelations by performing maximum pooling on the set of item-levelcorrelations row by row and column by column.
 6. The method of claim 1,wherein the answer is comprised in a sequence comprising a plurality ofordered answers with respect to the question, and determining thequality score comprises: determining a candidate quality score based onthe measurement of textual quality and the measurement of correlation;obtaining a reference quality score of an answer ranked before theanswer comprised in the sequence; determining a probability ofconverting the reference quality score to the candidate quality scorewith respect to the sequence; and determining the quality score based onthe candidate quality score and the probability determined.
 7. Themethod of claim 6, further comprising: determining the quality scoreusing a trained conditional random field (CRF) model.
 8. The method ofclaim 1, wherein extracting the question feature expression and theanswer feature expression comprises: extracting the question featureexpression using a first long-short term memory (LSTM) model trained;and extracting the answer feature expression using a second LSTM modeltrained, the second LSTM model being different from the first LSTMmodel.
 9. The method of claim 1, wherein extracting the question featureexpression and the answer feature expression comprises: obtaining afirst vectorized expression of the question and a second vectorizedexpression of the answer; extracting the question feature expressionbased on the first vectorized expression; and extracting the answerfeature expression based on the second vectorized expression.
 10. Anelectronic device, comprising: one or more processors; and a memory,configured to store one or more programs, wherein when the one or moreprograms are executed by the one or more processors, the one or moreprocessors are caused to extract a question feature expression of aquestion and an answer feature expression of an answer with respect tothe question, the question and the answer being represented in a form oftext; determine a measurement of textual quality of the answer based onthe answer feature expression; determine a measurement of correlation onsemantics between the question and the answer based on the questionfeature expression and the answer feature expression; and determine aquality score of the answer with respect to the question based on themeasurement of textual quality and the measurement of correlation. 11.The electronic device of claim 10, wherein the one or more processorsare caused to: divide the answer into a plurality of textual items, theanswer feature expression comprising feature elements corresponding tothe plurality of textual items, and the one or more processors arecaused to determine the measurement of textual quality by: determiningan importance of each textual item in context of the answer; anddetermining the measurement of textual quality by weighting the featureelements of the answer feature expression with the importance.
 12. Theelectronic device of claim 10, wherein the one or more processors arecaused to determine the measurement of correlation by: determining a setof item-level correlations between the answer and the question on alevel of textual item based on the answer feature expression and thequestion feature expression, the set of item-level correlationscomprising elements arranged in rows and columns, each element beingconfigured to indicate a correlation between a textual item of theanswer and a textual item of the question; and weighting the answerfeature expression and the question feature expression using the set ofitem-level correlations to determine the measurement of correlation. 13.The electronic device of claim 12, wherein the one or more processorsare caused to weight the answer feature expression and the questionfeature expression by: generating a first subset of significantcorrelations of the question with respect to the answer and a secondsubset of significant correlations of the answer with respect to thequestion by selecting an element having a higher value than apredetermined threshold row by row and column by column, from the set ofitem-level correlations; weighting the question feature expression usingthe first subset of significant correlations; weighting the answerfeature expression using the second subset of significant correlations;and linking the question feature expression weighted and the answerfeature expression weighted to obtain the measurement of correlation.14. The electronic device of claim 13, wherein the one or moreprocessors are caused to generate the first subset of significantcorrelations and the second subset of significant correlations by:generating the first subset of significant correlations and the secondsubset of significant correlations by performing maximum pooling on theset of item-level correlations row by row and column by column.
 15. Theelectronic device of claim 10, wherein the answer is comprised in asequence comprising a plurality of ordered answers with respect to thequestion, and the one or more processors are caused to determine thequality score by: determining a candidate quality score based on themeasurement of textual quality and the measurement of correlation;obtaining a reference quality score of an answer ranked before theanswer comprised in the sequence; determining a probability ofconverting the reference quality score to the candidate quality scorewith respect to the sequence; and determining the quality score based onthe candidate quality score and the probability determined.
 16. Theelectronic device of claim 15, wherein the one or more processors arecaused to: determine the quality score using a trained conditionalrandom field (CRF) model.
 17. The electronic device of claim 10, whereinthe one or more processors are caused to extract the question featureexpression and the answer feature expression by: extracting the questionfeature expression using a first long-short term memory (LSTM) modeltrained; and extracting the answer feature expression using a secondLSTM model trained, the second LSTM model being different from the firstLSTM model.
 18. The electronic device of claim 10, wherein the one ormore processors are caused to extract the question feature expressionand the answer feature expression by: obtaining a first vectorizedexpression of the question and a second vectorized expression of theanswer; extracting the question feature expression based on the firstvectorized expression; and extracting the answer feature expressionbased on the second vectorized expression.
 19. A non-transitory computerreadable storage medium having a computer program stored thereon,wherein when the computer program is executed by the processor, themethod for evaluating quality of an answer is implemented, the methodcomprises: extracting a question feature expression of a question and ananswer feature expression of an answer with respect to the question, thequestion and the answer being represented in a form of text; determininga measurement of textual quality of the answer based on the answerfeature expression; determining a measurement of correlation onsemantics between the question and the answer based on the questionfeature expression and the answer feature expression; and determining aquality score of the answer with respect to the question based on themeasurement of textual quality and the measurement of correlation. 20.The non-transitory computer readable storage medium of claim 19, whereinthe method further comprises: dividing the answer into a plurality oftextual items, the answer feature expression comprising feature elementscorresponding to the plurality of textual items, wherein, determiningthe measurement of textual quality comprises: determining an importanceof each textual item in context of the answer; and determining themeasurement of textual quality by weighting the feature elements of theanswer feature expression with the importance.